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ABSTRACT 


Most initial software problem resolution takes place through the 
Support Center, which is composed of people and computer systems 

in many locations around the country. This presentation, by a 

Level 1 PSR, will attempt to provide a thread through the labyrinth. 
I will describe the process of resolving your problems, what re- 
sources are available, how to get to them and who has responsibility 
for what system components. A sample APAR (Authorized Problem 
Analysis Report) will be explained. 


Permission is granted to SHARE, Inc. to publish this’presentation 


paper in the SHARE 61 precedings. IBM retains the right to dis- 
tribute copies of this presentation to whomever it chooses. 
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IBM SUPPORT CENTER 


Good afternoon, and once again welcome to SHARE. My name is 


Peggy Sykes. I am a Program Support Representative from the Tampa 


Support Center, where I assist customers with problem resolution 
via the telephone. 


. I will be discussing the different aspects of the IBM Sup- 
port Structure and the method of operation within each of these 
areas. The agenda consists of a support structure overview which 
focuses on the implementations of the center, what resources are 
available and how customers can better utilize these resources. 


Refer to Figure 1 


The Support Center concept was put into effect as a test in 
February of 1978. This was a result of increased complexity of 
teleprocessing networks and its applications. Because of the.wide 
usage of teleprocessing networks IBM addressed the problem of 
increasing system availability, and came up with the idea of remote 
support. 


Our test included 314 customers, consisting of DOS, MVS and 
VS1 users which lasted one year. During this time 2000 problems 
were addressed and our statistics revealed 65% were resolved by 
Level 1 without the involvement of any other segment within our 
support structure. We accomplished down time minimization for 
customers by providing an 800 number which provides access to our 
centralized service. . 


The technical reps are trained on all the System Control 
Programs (SCPs) and Program Products (PPs) we support. You dial 
an 800 number and receive PSR assistance almost instantly as op- 
posed to our previous method of arranging a convenient time with 
your local PSR. Customers also have better flexibility in sched- 
uling because we are available 24 hours a day. You are able to 
schedule maintenance, installation of new products and testing 
procedures during off hours or weekends when system availability 
is not as crucial. 


Figure 1 SUPPORT CENTER CONCEPT 
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Refer to Figure 3 
Refer to Figure 2 


The possible avenues a call can take are indicated here. 


The Support Center is comprised of a multitude of information, Customer initiates the call to Level 1. If the problem is a re- 
technical skill levels and administrative services. Listed are discovery it will be resolved by the Level 1 PSR with no further 
the benefits and the services provided. You are encouraged to interface from the other groups. However, if it is a new problem 
use these resources. Throughout the presentation I will expand or one that can not be resolved by Level 1 we will pass the call 
on each of these areas to ensure that you will be able to make the to Level 2. Unlike Level 1, who are the system specialists sup- 
Support Center work for you! porting a variety of products, Level 2 are component specialists 


with expertise in one particular area. At any time Level 1 and 
Level 2 are able to dispatch a PSR to go to the customers site if 
necessary. 


So you can see that we have not eliminated the field PSR, 
but made available an expedited support service in addition to 
your local help. 


There is also a change team, sometimes referred to as Level 3, 
for each Level 2 location. They specialize in correcting coding 










. errors in the Developers logic for their applicable component 
areas. 
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Refer to Figure 4 


When you encounter a problem, the first step should be prob- 
lem analysis. Determine what type failure you have. This is 
called problem determination, referred to as PD. Is it hardware 
or software? The customer can be assisted by the Customer Engineer 
in making this determination. The CE at all times has access to 
a PSR. If the problem is diagnosed as hardware, the CE will pur- 
sue its resolution. If it is software, further problem source 
identification is necessary to determine the failing module and 
source of the problem. PSI is the customers responsibility and 
if assistance is required, the Support Center has PSI queues for 
each SCP we support. 


To access this queue, when calling the Support Center, inform 
the Customer Service Coordinator (CSC) that you are uncertain of 
the failing component and would like the applicable PSI queue. 

For example VMPSI queue, MVSPSI queue, VSIPSI queue, etc.... These 
queues are managed by Senior PSRs who are technically capable. 

in assisting you to find the source of the problem or directing 

you in what additional documentation must be obtained to make this 

determination. The local PSR is also available for PSI assistance 

for problems that can not be isolated over the telephone. 
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There is also a manual called Programming System General In- 
formation (G229-2228-24) referred to as the PSGIM. It may be 
ordered through your Marketing Representative. It is intended 
for System Programmers or others who solve program problems at 
IBM customer accounts. It contains information about the IBM 
Support Center, what it is and how to use it, general information 
to aid in solving IBM program problems, and documentation and 
materials required to submit an APAR. This manual will also 
assist you with problem determination and problem source identi- 
fication guidelines. 


If the problem is not in one of the products handled by 
the Support Center we will refer you to the applicable product 
marketing representative. An example is ADRS. This is a Business 
Graphic feature supported by marketing. The Support Center is 
not staffed, nor do we have the required documentation to sup- 
port this product. We also do not provide sysgen guidance, direc- 
tions on how to set up JCL and product operation questions. 


In instances when you are uncertain of provisions offered by 
the Support Center, call in and ask for the Control desk. It is 
staffed by technical PSRs who will direct you to the correct re- 
sources. Products supported by the Support Center are listed in 
the PSGIM. 
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Refer to Figure 5 


We have three (3) Level 1 locations to serve our customers. 
The Tampa Center supports customers within our eastern regions, 
the Chicago Center handles the midwestern regions, and as of 
April 1, 1983, the new Boulder Support Center supports the west- 
ern regions. 


The support structure consists of Level 1 locations, Level 2 
groups, change teams, and local PSR. A commitment is made to 
meet the goal of quality service to every customer every time. 
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Refer to Figure 6 


Our three (3) level one locations enable us to balance our 
workload and provide us with back-up capabilities. We are avail- 
able for service 24 hours a day, 7 days a week, including all 
holidays. 


The Tampa Level 1 location handles prime shift for the east 
coast, and second and third shifts for the entire country. Our 
Level 2 centers work prime shift hours only, with the exception 
of priority 1 problems. Our second and third shift personnel 
have developed an increased expertise in problem solving which may 
enable them to provide a resolution that may normally be provided 
by Level 2. A lower call volume enables them to get more indepth 
in a problem. 


The Chicago Center rotates working holidays with the Boulder 
Center for prime shift only. We also have a Hardware Support Cen- 
ter for CE's and Data Link Library (DLL) connections which are 
housed in Chicago. 


Information Access (Info Access), which extends the capabil- 
ities of Info/Management and INFO/MVS by providing teleprocessing 
access to the IBM Customer Software Support Facility (CSSF) data 
base is maintained and supported through the Boulder Center. 
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Refer to Figure 7 


Our entire support structure uses an internal data base known 
as RETAIN. This stands for Remote Technical Assistance Information 
Network. There are two RETAIN systems, the primary system located 
in Tampa and the back-up system housed in Boulder. 


We run on twin 3033's and all information is instantly mir- 
rorred between the Tampa and Boulder systems. If the Tampa system 
goes down, at the flip of a switch we can run off the Boulder sys- 
tem. Rarely will both systems go down at the same time. If 
this situation ever occurs we will work on priority one calls, 
which are problems causing a critical impact to your environment. 
Other customers will be advised of an approximated system down 
time. 


RETAIN contains a vast amount of information and files. There 
are 4 files that Level 1 and Level. 2 basically use. Software Search 
Facility contains all APARs which are Authorized Program Analysis 
Reports. APARs contain a description of the software and hardware 
configuration, a synopsis of the problem symptoms, and is accom- 
panied by supporting documentation. If available, a source fix 
or superzap is included in an APAR as a corrective fix. 
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Figure 7 
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Also stored in SSF are records of Program Temporary Fixes, ref- 
fered to as PTFs. PTFs are complete module replacements that 
incorporate either one or more APAR fixes. 


Once PTF's are closed and ready for shipment, a Programming 
Update Tape (PUT) is built by Programming Service Support (PSS). 
The PUT tape will contain PTFs and their related information, 
such as, job streams, installation aids and special documentation. 
The Program Information Department (PID) will then ship customized 
PUT tapes according to products each customer is licensed for. 


The Problem Management System is also widely used. Within 
this file we create problem records and track all calls that come 
into the center. Since it is a searchable file, we can also find 
hits on user errors or problems that may not have required an APAR 
or PIP fix, 


The customer profile contains information about each customer. 
account we service, IBM representatives, mailing addresses, phone 
numbers and CPU data. The customer profile is used mainly to ver- 
ify customers mailing addresses when sending out install informa- 
tion. We will talk about how you can update this file later in the 
presentation. 


The last most widely used file is PSP, Preventive Service 
Planning. It contains all install and preventive service infor- 
mation that becomes available after a product has been released. 

For example, System Control Program (SCP) and Program Product (PP) 
exclude lists that document problems that are discovered after the 
product has been released. This information can be obtained by 
calling the Support Center and requesting install documentation by 
the applicable (FMID) Function Modification Identifier or tape num- 
ber. We will mail these BUCKETS (no one knows what this stands for, 
Other than a collection of information) to you in hardcopy format. 


VM370 customers can request PSP files of the current releas- 
es, to be mailed in hardcopy form or on tape. The tape will con- 
tain two files. The first file is a copy of the documentation, 
the same information, you would receive if requesting the hard copy. 
The second file is Control Program (CP) APARs in machine readable 
format. The actual source fixes are contained in the APARs. 


Earlier I mentioned how our telephone system provides us 
with back-up capabilities. We are able to transparently reroute 
calls anywhere in the country. Our phone system was designed for 
Support Centers usage by the telephone company. All Level 1 Sup- 
port Centers have sufficient telephone lines to handle our call 
volume. Therefore you should never receive a busy signal. If 
you encounter a problem with reaching the Support Center, contact 
your local branch office. 
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We are very sensitive to the amount of time a customer re- 
mains on hold. The call comes into the Communications Center and 
is placed on the appropriate extension until a technical rep be- 
comes available. There are call boxes located throughout the cen- 
ter and any time a customer has been on hold beyond five minutes, 
the system is designed to alarm that call. This is accomplished 
by a flashing light on the call box, and also an alarm bit being 
turned on in the RETAIN system. Our average hold time is approx- 
imately 2 minutes, this varies depending on the time of day. 
Generally, the majority of our calls come in around 10 o'clock in 
the morning and 2 o'clock in the afternoon. The telephone system 
1s also equipped to reroute calls between our three Level 1 loca- 
tions. 


To give you an example of our back-up capabilities, a few 
months ago the Chicago Center lost its 800 customer number. We 
were able to reroute all calls to the Tampa Center and for 24 hours 
we serviced the entire country. 


Refer to Figure 8 


The Tampa location is comprised of PSRs with an average of 
ten years IBM service and a three-year average of PSR experience. 


In addition to base training, which is approximately three 
months, education continues with an average of one month of 
training, per person, per year. Level 1 PSRs also gain tech- 
nical skills by attending internships, temporary assignments to 
Level 2 locations and informal seminars given at the Level 1 loca- 
tions. 


To enhance our communication abilities, management provides 
Communication Technique seminars to all Support Center person- 
nel. This produces conversation proficiency and courtesy, which 
are essential when conversing via the telephone. 


All Level 1 locations have extensive libraries containing 
documentation for everything we support. Module information is 
in microfiche form and manuals are accessible in hard copy for- 
mat. We are on an automatic shipping list from PID, our Program 
Information Department, allowing us to receive all documentation 
and updates. This allows us to keep current. The libraries are 
maintained and used daily by the PSRs. | 
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Refer to Figure 9 


Level 1 is the first and most important point of contact. 
This cannot be emphasized enough because it is our responsibility 
to analyze the problem and find any rediscoveries if possible. 
Each of the PSR's at Level 1 are system specialists on a select 
group of products and work with these products everyday. This, 
of course, breeds proficiency and the PSR is able to maintain a 
constant awareness of what is happening with those products. He 
or she is familiar with the most significant problems at any given 
level of the component. 


We service customers, PSRs, CEs, SEs: anyone with an access 
code which is also the customer eum s 


At Level 1 we will extract as.much Aromveien as possible 
and formulate a search argument to search the data base files 
for known problems. 


Level 1 also provides preventive and corrective service 
information taken from the PSP files. 


Figure 9 LEVEL 1 
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Refer to Figure 10 


Level 1 works in real time mode. When you call in, your 
call will be handled by the first available technical rep. No 
call back is required. We can do one of four things; resolve the 
problem, which is our main objective; pass the call to Level 2 
if it is a new problem or one that can not be resolved by Level 1; 
dispatch a PSR if onsite assistance is required; or request add- 
itional documentation if needed. 
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Refer to Figure 11 


Each location has an administration staff. . The Customer Ser- 
vice Coordinators, referred to as CSCs offload a considerable 
amount of the PSRs work, leaving them more time to assist in tech- 
nical problems. 


The Status queue is staffed by reps from our administrative 


group. Customers can bypass Level 1 PSRs and go straight to the 


status group to obtain problem status information, to re-queue to 
Level 2 with additional information, change the priority of a prob- 
lem, or to close a problem. 


The Branch Office Notification Desk (BOND) is used to alert 
the local branch office of priority one problems or any unique 
Situations that may arise within the accounts they are respons- 
ible for. 


The administration group is also responsible for printing 
and mail preparation of preventive service information. They 
dispatch PSRs and make temporary customer profile updates. If 
a customer is installing a new CPU, has changed addresses or phone 
numbers, the CSC rep working the PROFILE queue can change or add 
this information which will stay in the system 28 days. Any per- 
manent changes are made by the local branch office. 


Figure 11 ADMINISTRATION 
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Refer to Figure 12 


I have mentioned the term queue throughout the presentation. 
All Level 1 locations are subdivided into queue areas to separate 


us among the different products. For example, the VM queue, Data 


Management queue, PSI queue, etc... There is one Operations Man- 
ager per queue and one or two team leaders depending on how large 
the queue is. Team leaders are responsible for monitoring the calls 
to make sure they are taken in a timely manner and scheduling educ- 
ation. and vacation time. They also provide call assistance, when 
needed, to-PSRs in their queue area. 


The figure below shows the career path of PSRs that make up 
a queue. There are four skill levels of a PSR: Trainee, Associate, 
PSR and Senior PSR. The trainee, upon return from Base School will 
work in a controlled environment where call simulated training is 
provided. This additional training will last from six to eight 
weeks. The Associate PSR is gaining experience and technical 
skills to handle most calls. The PSR is the journeyman position 
with the ability to handle all calls. Senior PSRs will take or 
supervise PSI calls, provide technical advice to less experienced 
PSRs when required, in addition to normal call taking. 





Figure 12 LEVEL 1 QUEUE 
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Refer to Figure 13 


There are seven Level. 2 locations within the United States. 
Listed below are the product components. each location is respon- 
sible for. | 


The Development group, for the applicable product, is also 
located within the vicinity of each Level. 2 location. Generally, 
there is a change team (Level 3) group resident at each Level 2 
site. The cdéntiguity of the groups allows accessability inter- 
facing purposes when necessary. 


Although not shown on Figure 13, the Change Team for DOS/VSE 
products are located in Amsterdam, Boeblingen and Hursley. 


Figure 13 LEVEL 2 
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Refer to Figure 14 


Our Level 2 groups are the component experts. Unlike Level 1 
queues, that are subdivided ‘into product areas, Level. 2 will 
assign individual programmers and change team personnel to a com- 
ponent level. Working with these single elements everyday allows 
the responsible person to know his or her code thoroughly. There- 
fore, indepth problem analysis can be performed if your problem 
reaches Level 2. They are also. experienced in trouble-shooting 
dumps and- analyzing traces. 


If a new problem is discovered in the Developers logic of the 
System code, Level 2 is responsible for APAR preparation and its 
submission. They will work with the customer in gathering the 
necessary documentation and pursue a resolution. If you ever en- 
counter a new problem, the procedures for APAR submission are doc- 
umented in the PSGIM mentioned earlier in this presentation. After 
the APAR has been submitted, Level 2 will give the customer a cor- 
rective fix or circumvention until a PTF becomes available. Lev- 
el 2 is also responsible for the RETAIN Data Base, its accuracy 
and updates. 


Figure 14 | LEVEL 2 
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Refer to Figure 15 


Due to Level 2's methods of operation, they work in call back 
mode. Level 1 requeues the problem management record, which con- 
tains a problem description of the conversation between the Level 
1 PSR and the customer, to Level. 2 who will in turn call.the cust- 
omer back. At this point Level 2 owns the problem until its re- 
solution. They can do one of three things, search the data base 
for a rediscovery, APAR a new problem, request additional infor- 
mation or dispatch a PSR for on site assistance with the custom- 
er's concurrence. 
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Refer to Figure 16 


A time criteria has been established to permit expedited 
service when a call to the customer is required by Level 2. © 


Most important, priority of a problem is determined by the 
customer. You will know better than anyone else what impact a 
system problem is having on your organization. All incoming calls 
to the Support Center automatically default to priority 3. It is 
up to the customer to raise or lower the priority. 


Priority 1 indicates your system is down or the problem is 
Causing a critical impact to your system. Turnaround time objec- 
tive is one hour, no more than 2 hours. 


Priority 2 is when you are severely restricted due to the na- 
ture of the problem. The objective is to return these calls with- 
in four to six hours. 


Priority 3 states that you are able to circumvent the pro- 
blem and its impact is limited. You will generally receive a 
call within twenty-four hours or by the close of the next business 
day. 


Priority 4 lets us know no call back is required and only 


‘information is being passed on, such as to close a problem or 


to let us know a problem has been resolved. 


Level 2 locations work prime shift only, with the exception 
of priority 1 problems. Upon these occurrences a Level 2 rep 
will be contacted to work on the problem. Therefore, the custo- 
mer and his resources must also be available. All Level 2 loca- 
tions are'available during prime shift regardless of the customer's 
time zone. 


Sl ssesessensnusssumnssatstssstnstsaesensesessee 
Figure 16 LEVEL 2 
Caen nn a SSeS SSS a eetstnetmeemneset 


RESPONSE TIME CRITERIA 


PRT ORY A 
=) AOUR 


® PRIORITY 2 
- 4 HOUR 


PRIORITY 2 
> BY -CELSSe SF (NES BUSPNIESS: DAY 
PRIOR ee. 2 


- INFORMAT ID. ch 


e DETERMINED av cuUstrames 


GES 


Refer to Figure 17 


I have now covered every. possible avenue a call can take. 
Figure 17 summarizes what has previously been discussed. 


In addition to technical assistance, if procedural questions, 
concerns or complaints arise we have a control desk, which was ad- 
dressed earlier, available to service you. If escalation is re- © 
quired, every Level 1 and Level. 2 location has a duty manager avail- 
able twenty-four hours a day. 


ee a saa ara ls 
Figure 17 CALL FLOW 


Ta a ee 
CUSTOMER 


COMMUNICATION 


GEN TE:R 


cc 
ia ee 1 a { ee roy 4 
Sats SPR Oster. | es 
| MANAGEMENT | 7 
| RECORD ———— , 


rae a 4 = wert Ss ' E = 
GNTRO! ee Bee 
CON ROL /] ma Ro m™ 
: ame fe 


a ae anneal r- Se 


ww Le wy 














Refer to Figure 18 


Statistical information is tracked for all incoming calls to 
the Support Center. From this data we have extracted percentages 
of call distribution between Level 1 and Level: 2. 


The statistics below show the action taken on calls that came 
into the Support Center during December 1982 through March 1983. 


During these four months, Level 1 average completion rate was 
63.4%. Level 1 completed the call without the assistance of any 
other segment within our support structure. Calls passed to Level 
2 for the first time were 22.6%. Requeues, which are calls that 
have been to Level 2 and are going back because the customer was 
instructed to gather more documentation, or further problem analy- 
sis was necessary. The average number of requeues are 6.2%. On 
Site assistance, which required a PSR dispatch occurred 5.1%. An- 
other 2.7% of the calls were for miscellaneous data. For example, 
a call that may have been a product handled by the Support Center. 


Figure 18 CALL DISTRIBUTION LEVEL 1 (PRIME SHIFT) 
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The average time a PSR stays on the phone with a customer is 
approximately 22.4 minutes. This will vary depending on the na- 
ture of the call. If a call is for install information it may 
take five minutes, where a PSI call may take one hour. The time 
we stay on the phone with a customer is at the PSR discretion. 
The average time a customer. stayed on hold for the next available 
PSR was 1.9 minutes. 


Refer to Figure 19 


The following statistics are also for the Tampa Support Center, 
one week and prime shift only... Level 1 PSRs ‘handled 55% of the 
calls. Another 19% were housekeeping calls created by Tampa's 
APAR/PTE Quality Reps. They are responsible for alerting Level 2 
of any errors found in the RETAIN Data Base. Level 2 will then 


update or correct the problem. 


Branch office notifications by the BOND queue and PSR dis- 
patches totaled 6% of the call volume. This is a job handled by 
our Administrative staff, along with the final 20% status build 
calls. Therefore, you can see how our Administration group as- 
sists to lessens the PSRs' workload. This percentage includes 
closing calls, requeues to Level 2, status updates and changing 
call priorities. 





Figure 19 AVERAGE NUMBER OF CALLS CREATED PER WEEK 
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Refer to Firure 20 


Our Marketing Division consists of System Engineers (SE) and 
Marketing Representatives. . The Marketing Representative is respon- 
sible for sales and the SE assists customers in installation plan- 
ning and procedures. 


PSRs and CEs work in the Field Engineering Division. PSRs 
provide software support and the CE is responsible for hardware 
performance. 


In addition we have added the IBM Support Centers. This is 
an expedited software support service comprised of over 3,000 people 
across the country with a commitment to provide quality service 
to each of IBM's many customers. The development of remote support 
has proved to be an outstanding achievement for IBM. I sincerely 
hope the information I have provided will assist you in making the 
Support Center work for you! 
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Figure 20 
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